Repository navigation
fix(shortlist): drop option_order when a narrow no longer fits it - #1063
NandhaKishorM merged 2 commits into
Conversation
A choice narrowed by predict_shortlist or predict_tournament kept the caller's full-length option_order, which Agent._check_question then refuses: the order must be a permutation of range(n_options) and the kept label set is smaller. Nandha called this a core contract question on NandhaKishorM#958 and asked for the order to be dropped with a warning. These checks are red on main; they pin the narrowed question validating, one RuntimeWarning naming the question, an answer identical to the no-order call, and passthrough/non-choice questions untouched.
predict_shortlist narrowed criteria to the top-k labels while forwarding the caller's full-length option_order, and predict_tournament built its group and finalist questions the same way, so a choice carrying an order crashed in the agent: the order must be a permutation of range(n_options) and the kept label set is smaller. Nandha called this a core contract question on NandhaKishorM#958 and asked for the order to be dropped with a warning, since remapping it onto the kept labels would silently move options between slots. Both paths now drop the order on a narrowed question and emit one RuntimeWarning naming the question and the slot counts. Passthrough questions, non-choice questions and the caller's own dicts are untouched, so a call with an order answers exactly like the same call without one.
|
Merged. This follows the direction I gave on #958 exactly, including the part I cared most about. Dropping the order rather than remapping it onto the kept labels is the right call, and your reason is the right reason: remapping would silently move options between slots, so a caller who passed a deliberate order would get a different one back with no indication. A warning that names the question, the old slot count and the kept count tells them precisely what happened. Verified on the merged tree with a 6-option choice narrowed to 2: On Covering The unchanged list is the part that makes this safe to take: passthrough questions, non-choice questions, calls without an order, and the caller's own dicts all behave exactly as before, and a call with an order now answers identically to the same call without one. On your note about Now that core handles this, the refusal in All 86 suites pass, and the 82-row real-weights decision diff is unchanged. |
14 pull requests from 8 contributors. * the TypeScript tree parses again: an unterminated template literal shipped in 0.4.1 and stopped 22 of 34 test files loading (#1056) * x_jev_confidence on /v1/systemone, computed locally (#1066, #302) * a narrowed shortlist drops option_order with a warning instead of failing (#1063) * registering a name one edit from a built-in warns (#1055, #919) * compose forwards four knobs the runtime already read (#1053, #1054) * Java Router.predictBatch and Shortlist.predictTournament (#1060, #1061) * teacher-agreement evals, with the self-agreement ceiling (#1071, #919) Security: no malicious content found; the audit is recorded locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The wrapper has refused a narrowed choice with a full-length option_order since #958, because core then raised ValueError on it. #1063 changed core to drop the order instead of remapping it, so the refusal now blocks a call core can answer, and deleting it without a signal would silently lose the caller's deliberate order: MCP clients read JSON and never see core's RuntimeWarning about the drop. The tests demand that the narrowed call answers over the kept labels with a per-question option_order_dropped signal naming the old and kept option counts, visible through the JSON wrapper; no signal for passthrough, non-choice or order-free calls; an invalid order still refused as invalid_questions before any predict; the caller's dict unchanged and stdout clean, so the MCP JSON-RPC channel cannot be corrupted.
#958's refusal existed only because core used to crash on a narrowed question carrying a full-length order; #1063 made core drop the order rather than remap it onto the kept labels, so the MCP wrapper can now answer the call. Remove the blanket refusal -- validate_questions still rejects anything that is not a valid permutation before predict runs -- and report the drop in the response as option_order_dropped, per question, with the old and kept option counts. The signal is derived from the original question and the returned shortlist meta, never from captured warnings, which are process-global and could attribute another call's drop to this one. Passthrough (n <= k), non-choice and order-free responses are what they were: the key appears only when an order was actually dropped. The tool description, docstring and docs/cli-mcp.md name the new response key.
What
predict_shortlistnarrows a choice'scriteriato the top-k labels while forwarding the caller's full-lengthoption_order, andpredict_tournamentbuilds its group and finalist questions the same way incut(). The narrowed question then failsAgent._check_question, because the order must be a permutation ofrange(n_options)and the kept label set is smaller. Reproduced on main1adc59f7:Fix
Following the direction you gave on #958, a narrowed choice now drops its
option_orderwith oneRuntimeWarningnaming the question, the old slot count and the kept label count. The order is not remapped onto the kept labels, since that would silently move options between slots.Both paths are covered:
predict_shortlist, andcut()inpredict_tournament(the round groups and the finalists share that choke point, so one helper covers them). The warning fires once per question per call, not once per tournament round.Unchanged: passthrough questions (n <= k), non-choice questions, the caller's own dicts (both paths copy before editing), and calls without an
option_order— those emit no warning and behave as before, so a call with an order answers exactly like the same call without one. Verified on real weights for both paths.Tests
python tests/test_shortlist.py→ 219 passed, 0 failed (on main: 209 passed, 10 failed — the 10 new checks)python tests/test_hooks_api.py→ 572 passed, 0 failedruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120clean;python -m compileall -q laya/ tests/exit 0tests/test_mcp.pycould not run in my environment: the installedmcpis 1.29.1 and the suite requires >= 2.2.0. It fails the same way on unmodified main, so this is an environment block, not a result — and MCP behaviour is unchanged, since fix(mcp): keep and validate a question's option_order #958 already makeslaya_shortlistrefuse a narrowed question that carries an order.Two commits: the regression tests first (red on main, green with the fix), then the fix itself.
Closes #1062